Camera using multiple lenses and image sensors to provide improved focusing capability

ABSTRACT

An electronic camera for producing an output image of a scene from a captured image signal includes: (a) a first imaging stage comprising a first image sensor for generating a first sensor output; a first lens for forming a first image of the scene on the first image sensor; and a first lens focus adjuster for adjusting focus of the first lens responsive to a first focus detection signal; and (b) a second imaging stage comprising a second image sensor for generating a second sensor output; a second lens for forming a second image of the scene on the second image sensor; and a second lens focus adjuster for adjusting focus of the second lens responsive to a second focus detection signal. A processing stage either (a) selects the sensor output from the first imaging stage as the captured image signal and uses the sensor output from the second imaging stage to generate the first focus detection signal for the selected imaging stage, or (b) selects the sensor output from the second imaging stage as the captured image signal and uses the sensor output from the first imaging stage to generate the second focus detection signal for the selected imaging stage.

FIELD OF THE INVENTION

The present invention relates to a digital camera that produces digitalimage files and, more particularly, to a digital camera that usesmultiple lenses and image sensors to provide improved focusingcapability.

BACKGROUND OF THE INVENTION

Currently, most digital cameras use a zoom taking lens and a singlecolor image sensor to capture still and motion images. The capturedimages are then processed to produce digital image files, which arestored in a digital memory in the camera. The digital image files canthen be transferred to a computer, displayed, printed, and shared viathe Internet.

In order to capture sharp images of moving subjects, a digital cameraneeds to provide a precise automatic lens focusing system (i.e., anautofocus system). The autofocus system must be capable of quicklyobtaining the correct focus in order to minimize the “shutter delay”between the time the shutter button is pressed and the still image iscaptured. The autofocus system must also work in a continuous imagecapture mode wherein video images are captured. For instance, in a videomode the focus should be adjusted in real-time while video images arebeing continuously captured.

Many digital cameras and scanners capture images using an image sensorand a taking lens system with an adjustable focus. Typically, the focusdistance of such an adjustable focus taking lens system canautomatically be set to one of a plurality of different settings bysensing, control, and drive systems, which are adapted to provideoptimal focus of what is determined to be a subject area in a scene.Lens systems that provide automatically adjustable focus settings basedon a focus measurement and an adjustable focus lens are referred toherein as autofocus systems. Digital cameras typically use one of twotypes of autofocus systems: a rangefinder system and a“through-the-lens” focus system.

A rangefinder system uses rangefinding sensors such as a sonicrangefinder or a dual lens rangefinder to determine the distance from acamera to one or more portions of a scene within a field of view of therangefinder system. A sonic rangefinder measures the phase offsetbetween a projected sonic signal and a reflected sonic signal to inferthe distance to objects in the scene. Dual lens rangefinders contain twolenses that are separated by a distance along with two matching sensorareas that capture matched pairs of images. Dual lens rangefinders arecommonly used on digital cameras in the form of dual lens rangefindermodules which contain two lenses separated by a distance along with twomatching sensor areas that capture matched pairs of low resolutionimages.

Common dual lens rangefinder-based autofocus systems include active andpassive systems. Active systems actively project light onto the scene,while passive systems work with the available light from the scene. Duallens rangefinder modules can be purchased from Fuji Electric in severalmodels such as the FM6260W. A dual lens rangefinder module for opticalapparatus such as a camera is described in U.S. Pat. No. 4,606,630,which was issued to Haruki et al. on Aug. 19, 1986 (and assigned to FujiElectric). According to the description of the prior art in this patent,matched pairs of low resolution images are analyzed for correlationbetween the two images to determine the offset between the two imagescaused by the separation between the two lenses.

A diagram illustrative of the principle of the operation of aconventional rangefinder is shown herein in FIG. 27. In that diagram,light from an object 151 is incident on two small lenses 152 and 153which have a sufficiently short focal length f that light rays receivedfrom the object through different spaced paths 154 and 155 producecorresponding spaced images 157 and 158 in a focal plane 156 which iscommon to the lenses 152 and 153. When the object 151 is at an infinitedistance, the centers of the images 157 and 158 are located at referencepositions 170 and 180 in FIG. 27, but when the object 151 is located ata closer distance, the centers of the images are shifted apart topositions 171 and 181. If the distance by which the images 157 and 158are shifted from the reference positions 170 and 180 are designated x₁and x₂, respectively, then the total shift x may be expressed asfollows:

x=x ₁ +x ₂ =b·f/d

Thus, the distance d to the object 151 can be measured by d=b·f/x. Inthis case, b is the distance between the optical axes of the smalllenses, that is, the base length. To obtain the shifted amounts x₁ andx₂, or the sum x of both, two optical sensor arrays 190 and 191 areprovided in the focal plane 156 as shown in FIG. 27. These opticalsensor arrays each comprise a plurality of optical sensors, for instanceCCD devices, and an analog photoelectric signal is generated by eachoptical sensor corresponding to the light intensity at the portion ofthe image which is incident on the sensor. Haruki et al. shows aconventional circuit, as well as a higher speed rangefinding circuitaccording to the patent, for obtaining the sum x of the shifteddistances by comparing two image signal trains comprising the digitalimage signals from the left and right optical sensor arrays.

Basically, the offset information x is used along with the lensseparation distance b and the focal length f to calculate the distancedto the scene by triangulation. The calculated distance d to the scene isused to guide the positioning of an adjustable focus lens to produce thebest image quality. As known in the prior art, this adjustment may bebased on a calibration curve established between the distance to thescene as measured by the dual lens rangefinder module and a series ofbest focused images as produced by a “through the lens” autofocussystem. The calibration curve is stored as an equation or a look-uptable in a microprocessor in the camera.

Rangefinder-based autofocus systems have the advantage of being veryfast, some having a response time that can be in the range of 0.01-0.05second. However, the focus quality produced by some rangefinder-basedautofocus systems can vary when they are used in different operatingconditions. For example, sonic autofocus systems cannot focus through aglass window as the glass stops the projected sonic signal, therebycausing the autofocus system to focus onto the glass. In the case of adual lens rangefinder autofocus system, the accuracy of dual lensrangefinders are typically influenced by changes in environmentalconditions such as temperature and/or humidity. The problem with duallens rangefinder modules is that the calibration between the dual lensrangefinder module and the adjustable focus lens position is not stablewithin the normal operating environment for digital cameras.Environmental conditions such as changes in temperature and humidity cancause the distance to the portion of the scene as measured by the duallens rangefinder module to change by over 10%. In addition, the measuredposition of the adjustable focus taking lens in the adjustable focustaking lens system is prone to environmentally induced changes as wellso that inaccuracies are produced in the control system for theadjustable focus lens. Consequently, dual lens rangefinder modules arenot typically used independently for autofocus in digital cameras butinstead are used as a rough focus adjustment that is supplemented by a“through the lens” autofocus system.

Alternatively, the “through-the-lens” autofocus system determines afocus state through an analysis of a series of autofocus images capturedwith the adjustable focus lens system positioned at a plurality ofdifferent focus distances. For example, in a typical “through-the-lens”autofocus system a plurality of autofocus images (e.g., 5-20) arecaptured with the adjustable focus lens in a series of differentpositions in a so-called “hill climb” method. This type of autofocus isknown as “hill climbing” autofocus because it generates a sequence ofvalues that increase in level until they pass over a peak, i.e., a“hill”. In other words, the lens focus position is adjustedautomatically until the contrast of the edge detail in the image, or aparticular area of the image, is maximized. For instance, the contrastpresent in each of the autofocus images is compared and the autofocusimage with the greatest contrast is deemed to have been captured withthe best focus conditions (often the best focus lens position is furtherrefined by interpolating the contrast values between images).

In order to decrease focusing response time without sacrificing focusingprecision, it is common to use filters to separate not only the higherfrequency component of the video signal, but also the lower frequencycomponent. For example, a lens may be quickly driven in coarseadjustment steps in a low frequency range furthest from the maximumfocus, and then driven in finer adjustment steps in a high frequencyrange nearer to the maximum focus. A flow diagram of a conventional“hill climbing” contrast autofocus algorithm is shown in FIG. 28. Thisalgorithm uses the “hill climbing” contrast autofocus method discussedabove and shown in the diagram of FIG. 29, which illustrates therelationship between the focus value obtained from the filters and thelens position. In FIG. 29, the abscissa indicates the focusing positionof a lens along a distance axis, the ordinate indicates the focusingevaluation value, and the curves A and B indicate the focusingevaluation values for high and low frequency components, respectively,relative to a particular in-focus position P.

Referring to the flow diagram of FIG. 28, the best starting point forthe algorithm depends on the hyperfocal distance of the current lenssetting, which is a function of the focal length setting and thef-number. A distance of about 2 meters is typically a good startingpoint. Then a low frequency bandpass filter is loaded (stage 197) andthe focus values are read out. The algorithm employs a comparison stage198 to set the direction of lens adjustment toward increasing focusvalues, and to determine when the lens is stepped over the “hill”. Thedepth of field, which depends on the present focal length and f-number,sets the number of steps, i.e., the next near focus position, whichshould be taken before capturing the next frame when using the lowfrequency bandpass filter. Once the peak of the hill is passed (curve Bin FIG. 29), a high frequency bandpass filter is loaded (stage 199), andthe lens is moved in the opposite direction until the peak of the higher“hill” is found (curve A in FIG. 29). The peak focus value may useeither the weighted average or peak value from numerous pixels.

“Through-the-lens” autofocus systems are very accurate since theymeasure focus quality directly from autofocus images captured with thehigh quality taking lens. Unfortunately, “through-the-lens” autofocussystems can be relatively slow in determining a focus setting due to thelarge number of autofocus images that must be captured and compared. Forexample, “through-the-lens” autofocus systems can take as long as0.5-2.0 seconds to determine focus conditions.

Accordingly, in some digital cameras, the two types of autofocus systemsare used together in a hybrid system in which the rangefinder basedautofocus system is used to provide a fast estimation of the adjustablefocus lens location that is then followed by the use of the“through-the-lens” autofocus system to refine the focus setting. Forexample, U.S. Pat. No. 6,864,474, entitled “Focusing Apparatus forAdjusting Focus of an Optical Instrument” and which issued Mar. 8, 2005in the name of Misawa, describes the coordinated use of arangefinder-based autofocus system with a “through-the-lens” autofocussystem. In Misawa, the focus position of the adjustable focus takinglens is determined by both the rangefinder-based autofocus system andthe “through-the-lens” autofocus system. The difference between theadjustable focus taking lens position determined by therangefinder-based autofocus system and the adjustable focus taking lensposition determined by the “through-the-lens” autofocus system is storedfor future reference. In subsequent image capture episodes, the storeddifference information is used to refine the number of autofocus imagescaptured and analyzed by the “through-the-lens” autofocus system in the“hill climb” method to determine the adjustable focus lens position forbest focus, thereby reducing the number of autofocus images captured andprocessed in cases where the rangefinder system is accurate andincreasing the number of autofocus images captured and processed incases where the rangefinder is inaccurate. However, the method describedby Misawa assumes that the performance of the rangefinder, adjustablefocus taking lens system and control system are consistent over time, donot fluctuate with variations in environmental conditions, and do nototherwise change or drift over time.

Once an image is in focus, the “hill climb” method typically operatesover incremental distances near the subject presently focused upon.Then, in refocusing an image, the “hill climb” method typicallydetermines whether any lens movement is stepping “up or down the hill”and resets the lens for a new maximum. In practice, this means that, ifthe lens movement is stepping “down the hill”, the lens motion isimmediately reversed so as to seek the new maximum for the existingsubject. This is a particular problem in video focusing, where a newsubject at some distance away from the present subject may come into theimage and never be detected by the “hill climb” method—even where thenew subject may present a greater “hill” in terms of contrast values.One way of responding to this problem is referred to as “whole way”autofocusing, where the auto focus module looks over all the distancesdiscernible by the taking lens before deciding upon a focus position.

Commonly assigned U.S. Pat. No. 6,441,855 describes a “whole-way”autofocusing method, where a focusing device includes a movable focusinglens adapted to be moved to different positions across the entirefocusing range, a conversion element for converting light incident onand transmitted through the focusing lens into a signal, and a lensdriving mechanism for moving the focusing lens. The focusing devicefurther includes a focus evaluation value calculation unit forcalculating a focus evaluation value for each position of the focusinglens based on the signal from the conversion element. The focusevaluation value calculation unit extracts only the signalscorresponding to the pixels in a focus area defined, e.g., at the centerof an image, which is further divided into nine “tiles”, that is, blocksthat are obtained by dividing the focus area into a small number of rowsand columns used as observation areas.

In calculating the focus evaluation values, a determination is firstmade as to whether or not the calculation of the focus evaluation valueshas been repeated a certain number of times, e.g., ten times, fordifferent distance settings. When the determination is negative, thefocusing lens is moved by a preset step width, and the calculation isrepeated. Thus, the focusing lens is always moved stepwise from aninfinite far position to a nearest position, and for each step ofmovement, a focus evaluation value is calculated for each tile. Thesecalculations are performed for the respective tiles, to thereby obtainthe focus evaluation values for ten lens positions for each of the ninetiles, including all of the peaks that are found across the totaldistance. Using the ten total sums obtained for the respective lenspositions as the focus evaluation values, the focusing lens positionproducing the maximum peak is determined as the in-focus lens position.A lens driving output is then applied to the lens driving mechanism sothat the lens moves to the determined in-focus position.

In order to provide a small size digital camera with a large “opticalzoom range”, a digital camera can use multiple image sensors withdifferent focal length lenses, as described in commonly assigned U.S.patent application Ser. No. 11/062,174, entitled “Digital Camera UsingMultiple Lenses and Image Sensors to Provide an Improved Zoom Range”,which was filed Feb. 18, 2005 in the names of Labaziewicz et al., thedisclosure of which is incorporated herein by reference. For example,the Kodak Easyshare V610 dual lens digital camera includes a 38-114 mm(35 mm equiv.) f/3.9-f/4.4 lens and a 130-380 mm (35 mm equiv.) f/4.8lens, in order to provide a 10× optical zoom range. However, in boththis patent application and product, only one of the two image sensorsis used at a time. The two image sensors do not simultaneously captureimages.

U.S. Patent Application Publication No. US 2003/0020814, which waspublished Jan. 30, 2003, discloses an image capturing apparatus having aplurality of capturing optical systems, each coupled to a CCD imagesensor, including a first system having a shorter focal length and asecond system having a longer focal length. In the various embodimentsdescribed in this disclosure, the two lenses can provide different focallengths ranges, including one system with a fixed-focus lens and theother system with a zoom lens, or they can both be fixed focus lensesset to two different focus distance settings. In each case, rather thanobtaining user input, a selection unit automatically selects the capturesignals from one of the capturing optical systems based on captureconditions, such as measured distance or luminance, determined by acapture condition acquiring unit. The autofocus for these systems isprovided using a separate distance sensor. Neither of the two CCD imagesensors are used for the autofocusing operation.

U.S. Patent Application Publication No. US 2003/0160886, which waspublished Aug. 23, 2003, discloses a digital camera having twophotographing systems that are independent of each other. One embodimentshows one system including a monofocal “ordinary mode” lens and theother system including a zoom “telescopic mode” lens, each generating animage. An operator-actuated change over switch determines which image isto be recorded. Autofocus is also disclosed in connection with theseparate photographing systems, where a “hill-climb” contrast comparisontechnique used in one system complements a “hill-climb” contrastcomparison technique used in the other system. When it is desired tocapture an image from the telescopic mode optical system, a roughautofocus search (where a stepping motor may be driven at intervals ofseveral steps) is made by the ordinary mode optical system (where thefocal depth is relatively large). This rough search results in a reducedfocal distance range that includes the focusing position. Using thefocal distance range information provided by the ordinary mode opticalsystem, the telescopic mode optical system is driven to an autofocussearch start position at one end of the reduced focal distance range.Then, a fine autofocus search is performed by the telescopic modeoptical system (where the focal depth is relatively shorter), but onlyin the reduced focal distance range determined by the ordinary modeautofocus search. (When it is desired to capture an image from theordinary mode optical system, the autofocus search is made solely by theordinary mode optical system, with the telescopic mode optical systemplaying no part in the autofocus search.)

In another embodiment in U.S. Patent Application Publication No. US2003/0160886, which does not depend on the rough vs. fine searchmentioned above, a “hill climb” contrast comparison search is performedwhile the focusing lens of a first optical system is driven stepwise soas to move from an infinite distance setting toward a closest distanceposition, and a second “hill climb” contrast comparison search isperformed while the focusing lens of a second optical system is drivenstepwise from the closest position toward the infinite setting. Thisprocedure continues until a maximum contrast position is located,although neither system ordinarily needs to move through its entirerange. This tends to reduce the time period for detecting a focusingposition. In this embodiment, each of the optical systems could be usedfor capturing an image and for focus adjustment, or one optical systemcould be employed for capturing an image and focus adjustment and theother optical system could be devoted only to focus adjustment of theimage-capturing optical system. In another embodiment, in the case wherethe non-capturing optical system determines the focusing position first,the capturing optical system is driven to that position and a fineadjustment is then made by the capturing optical system.

A problem with these prior art systems is that either a separateautofocus sensor must be used (thus increasing the cost) or else thereis typically a significant “shutter delay” as the autofocus is performedusing the same sensor that is used to capture the image. Moreover, theseparate autofocus sensor is usually a rangefinder and, as mentionedabove, the calibration between the dual lens rangefinder module and theadjustable focus lens position is not stable within the normal operatingenvironment for digital cameras. Where the autofocus is performed withthe “through-the-lens” taking system, the process can be relatively slowin determining a focus setting due to the large number of autofocusimages that must be captured and compared. The problem can be somewhatalleviated according to the aforementioned U.S. Patent ApplicationPublication No. US 2003/0160886, but difficulties remain in rapidlyachieving focus as the subject changes or moves, or in rapidlyinterchanging the focusing requirements of the optical systems when theoperator elects to change the capture function from one photographingsystem to the other.

A special problem arises during video capture, where the autofocusimages are derived from the same series of still images or frames thatcompose the video images. Consequently, the process of autofocusing maycause 5-20 or more out of focus video images to be produced in the videoeach time the scene changes. As a result, during video capture with panmovements of the camera where the scene changes continuously, largeportions of the video are actually out of focus as the autofocus systemhunts for proper focus. A further problem is that during video capture,many of the frames are out of focus due to the use of an autofocussystem that uses the “hill climb method” to focus.

What is therefore needed is a digital camera that provides precise,rapid autofocus in both still and video modes without unduly increasingthe size or cost of the digital camera.

SUMMARY OF THE INVENTION

The object of this invention is to provide an improved autofocuscapability in a multi-lens digital camera without unduly increasing thesize or cost of the camera.

Another object of the invention is to provide an improved autofocuscapability for both still and video images.

A further object of the invention is to provide a measure of the changein focus conditions during the capture of video images that can be usedto control the refocusing operation.

A further object of the invention is to provide a measure of a change inthe distance to the scene for the purpose of controlling a refocusingoperation.

The present invention, which is directed to overcoming one or more ofthe problems set forth above, pertains to an electronic camera forproducing an output image of a scene from a captured image signal.Briefly summarized, the invention comprises, according to a firstembodiment of the invention:

a first imaging stage comprising a first image sensor for generating afirst sensor output, a first lens for forming a first image of the sceneon the first image sensor, and a first lens focus adjuster for adjustingfocus of the first lens responsive to a first focus detection signal;

a second imaging stage comprising a second image sensor for generating asecond sensor output, a second lens for forming a second image of thescene on the second image sensor, and a second lens focus adjuster foradjusting focus of the second lens responsive to a second focusdetection signal; and

a processing stage for either (a) selecting the sensor output from thefirst imaging stage as the captured image signal and using the sensoroutput from the second imaging stage to generate the first focusdetection signal for the selected imaging stage or (b) selecting thesensor output from the second imaging stage as the captured image signaland using the sensor output from the first imaging stage to generate thesecond focus detection signal for the selected imaging stage, wherebythe focus detection signal is applied to the lens focus adjuster of theselected imaging stage in order to adjust the focus of the imageproviding the sensor output for the captured image signal.

Particularly in relation to a video camera for producing an output videoimage of a scene from a captured image signal, the invention furthercomprises, according to a second embodiment of the invention:

a first imaging stage comprising a first image sensor for generating afirst sensor output, a first lens for forming a first image of the sceneon the first image sensor, and a first lens focus adjuster forcontinually readjusting focus of the first lens responsive to a focuschange detection signal;

a second imaging stage comprising a second image sensor for generating asecond sensor output and a second lens for forming a second image of thescene on the second image sensor; and

a processing stage for using the sensor output from the first imagingstage as the captured image signal and producing the output video imageof the scene, said processing stage using the sensor output from thesecond imaging stage to detect a change in focus of the video image ofthe scene and to generate the focus change detection signal for thefirst imaging stage, whereby the focus change detection signal isapplied to the lens focus adjuster of the first imaging stage in orderto continually readjust the focus of the first image providing theoutput for the captured image signal.

Briefly summarized, the invention generally comprises the use of two (ormore) image capture stages wherein an image capture stage is composed ofa sensor, a lens and a lens focus adjuster, in a multi-lens digitalcamera in which the two (or more) image capture stages can be used to:separately capture images of portions of the same scene so that oneimage capture stage can be used for autofocus while the other(s) is usedfor capturing a still image or a video; or alternately, the imagescaptured by the two (or more) image capture stages can be compared toeach other to measure the distance to portions of the scene bytriangulation for purposes of autofocus control.

In thus utilizing the invention, a digital camera advantageouslyprovides precise, rapid autofocus in both still and video modes withoutunduly increasing the size or cost of the digital camera.

These and other aspects, objects, features and advantages of the presentinvention will be more clearly understood and appreciated from a reviewof the following detailed description of the preferred embodiments andappended claims, and by reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a block diagram of a first embodiment of a digital camerausing a first zoom lens with a first image sensor, and a second zoomlens with a second image sensor according to the invention.

FIGS. 2A and 2B are two perspective views of the digital camera shown inFIG. 1.

FIG. 3 depicts a flow diagram showing a method for performing autofocusand for capturing digital still images according to a first embodimentof the invention using the digital camera shown in FIG. 1.

FIG. 4 depicts a flow diagram showing a method for performing autofocususing a rangefinder method with two image capture stages.

FIG. 5 depicts a flow diagram showing a method for performing autofocususing a “hill climb method” with two image capture stages.

FIG. 6 depicts a flow diagram showing a method for producing anautofocus rangefinder calibration curve.

FIG. 7 depicts a flow diagram showing a method for producing anautofocus “hill climb method” calibration curve.

FIG. 8 depicts a flow diagram showing a method for performing autofocusand for capturing digital video images according to a second embodimentof the invention using the digital camera shown in FIG. 1.

FIG. 9 depicts a flow diagram showing a method for performing autofocuswith a rangefinder method with two image capture stages.

FIG. 10 depicts a flow diagram showing a method for performing autofocuswith a “hill climb method” with two image capture stages.

FIG. 11 depicts a method for producing a range map with two imagecapture stages.

FIG. 12 depicts a block diagram of a digital camera using a first fixedfocal length lens with a first image sensor, and a second (zoom) lenswith a second image sensor according to another embodiment of theinvention.

FIG. 13 depicts a block diagram of a digital camera using a first fixedfocal length lens with a first image sensor, a second (zoom) lens with asecond image sensor, and a third (zoom) lens with a third image sensoraccording to another embodiment of the invention.

FIG. 14 depicts a flow diagram showing a method for enhancing the depthof field of an image by using images from both image capture stages.

FIG. 15A and FIG. 15B depict a diagram of a mobile phone camera with twoimage capture stages according to another embodiment of the invention.

FIG. 16A and FIG. 16B depict a diagram of a stage containing two imagecapture stages in a single stage for a mobile phone camera.

FIG. 17A and FIG. 17B are representations of images captured with twoimage capture stages showing the offset between the two images that isused to determine the distance to portions of the scene.

FIG. 18 depicts a flow diagram of a method for determining GPS locationsof portions of the scene.

FIG. 19 depicts a flow diagram for selecting one of the imaging stagesin a dual lens camera system as the primary capture unit, whilerelegating the other imaging stage to other functions, such as sceneanalysis.

FIG. 20 depicts a flow diagram illustrating the usage of the non-primarycapture unit for scene analysis.

FIG. 21 depicts a flow diagram illustrating the usage of the non-primarycapture unit for scene analysis whenever the change in scene conditionsexceeds a threshold.

FIG. 22 depicts a flow diagram illustrating the usage of both theprimary capture unit and the non-primary capture unit for sceneanalysis.

FIG. 23 depicts a flow diagram illustrating the reversal of thefunctions of the capture units, that is, the current scene analysis andprimary capture units are reset to be the primary capture unit and sceneanalysis capture unit, respectively.

FIG. 24 depicts a flow diagram illustrating the usage of a preview imagetaken by the primary capture stage to set the capture unit parametersfor the primary capture stage, specifically during the capture process.

FIG. 25 depicts a flow diagram illustrating the usage of a preview imagetaken by the primary capture stage and a scene analysis image taken bythe non-capture stage to set the capture unit parameters for the primarycapture stage, specifically during the capture process.

FIG. 26 depicts a flow diagram illustrating the usage of a primary imagetaken by the primary capture stage and an augmentation image taken bythe non-capture stage to produce an enhanced image.

FIG. 27 depicts a diagram illustrative of the operation of aconventional dual lens rangefinder.

FIG. 28 depicts a flow diagram of a conventional “hill climbing”contrast autofocus algorithm.

FIG. 29 illustrates the relationship between the focus value obtainedfrom the filters used to isolate components of the image and the lensposition for a “hill climbing” autofocus method.

DETAILED DESCRIPTION OF THE INVENTION

Because digital cameras employing imaging devices and related circuitryfor signal processing are well known, the present description will bedirected in particular to elements forming part of, or cooperating moredirectly with, apparatus in accordance with the present invention.Elements not specifically shown or described herein may be selected fromthose known in the art. Certain aspects of the embodiments to bedescribed may be provided in software. Given the system as shown anddescribed according to the invention in the following materials,software not specifically shown, described or suggested herein that isuseful for implementation of the invention is conventional and withinthe ordinary skill in such arts.

Each of the several embodiments of the invention described hereininclude an image capture assembly, such as a digital camera—still orvideo—or a digital scanner, having multiple image capture stages, eachcomposed of a lens and an image sensor, wherein the lenses of themultiple image capture stages have different focal lengths to provide anextended optical zoom range for the image capture assembly. The presentinvention contemplates the use of the multiple image capture stages toadditionally provide an enhanced autofocus capability. By using theimage capture stages for image capture and autofocus, dedicatedautofocus modules can be eliminated thereby reducing the cost and sizeof the image capture assembly while improving the sharpness of thecaptured still and video images, as well as increasing the speed ofresponse of the autofocus system.

There are several embodiments of the invention by which one imagecapture stage may be used to capture digital still images or videoimages while another image capture stage is simultaneously being usedfor another purpose, such as enhanced autofocus, generation of asecondary image, production of a range map, and the like. In a firstembodiment of the invention described herein, when a user has set a zoomposition to be within a first optical zoom range, a first imaging stagecontaining a first (e.g., zoom) lens is used to capture a still image ora series of images as in a video sequence, while a second imaging stagesimultaneously provides images for the purpose of autofocus of the firstimaging stage. Since the second imaging stage is not used to capture theimages, the focus conditions of the lens in the second imaging stage canbe adjusted over a wider range, e.g., around a peak contrast position(e.g., “hill climb” autofocus) or from the near focus position to theinfinity focus position (e.g., “whole way” autofocus) to determine thenew best focus setting for the lens in the first imaging stage withoutdetrimentally affecting the focus quality of the images captured by thefirst imaging stage. When the new best focus condition has beendetermined using the second imaging stage, the focus condition of thefirst imaging stage is changed from the previous best focus condition tothe new best focus condition.

When the user adjusts the zoom position on the image capture assembly toa second optical zoom range, the camera automatically switches to usethe second imaging stage (containing, e.g., a second zoom lens) tocapture the images, and begins using the first imaging stage forautofocus of the second imaging stage. Because the two lenses havedifferent focal lengths, they have different magnifications. Therefore,the digital filters used to determine the autofocus may be adjusted, asa function of this difference in magnification, in order to compensatefor the differences in magnification.

In a variation of this embodiment of the invention, two image capturestages are used together to form a high resolution rangefinder similarto a dual lens rangefinder but with higher resolution, which is providedby the two high resolution image capture stages and a larger separationdistance between the two lenses in the two image capture stages. In thisvariation, the first image capture stage may be used to provide aninitial accurate autofocus using a “hill climb” contrast comparisonmethod; subsequently, the first image capture stage is used inconjunction with the second image capture stage as a high resolutionrangefinder operating in a differential mode to detect any changes inthe distance to the scene that require refocusing prior to capture of adigital still image or prior to or during a video capture. By using therangefinder in a differential mode, that is, to discern a change indistance from an already focused position to a nearby changed focusposition, the impact of environmental changes on the accuracy of therangefinder is diminished.

In another embodiment of the invention described herein, the two imagecapture stages are both autofocused at the same time using a “hillclimb” contrast comparison method prior to capture of a digital stillimage or capture of a video image by the first image capture stage. Thesecond image capture stage then continues to check the focus bymeasuring the contrast in the image; when a change in contrast isdetected, a second autofocus operation using the “hill climb” contrastcomparison method is performed using the second image capture stage todetermine the change in focus condition. The focus of the first imagecapture stage is then changed by an amount proportional to the change infocus determined by using the second image capture stage. Again, adifferential focus change measurement is performed from a positionestablished by a “hill climb” autofocus to improve the accuracy of theautofocusing process.

As mentioned in the background of the invention section, a specialproblem arises during video capture, where the autofocus images arederived from the same series of still images or frames that compose thevideo images. For instance, during video capture with pan movements ofthe camera where the scene changes continuously, large portions of thevideo are actually out of focus as the autofocus system hunts for properfocus. Moreover, many of the frames may be out of focus due to the useof an autofocus system that uses the “hill climb method” to focus, whichas mentioned earlier may be unable to discern changing focus undercertain conditions where the subject of interest has suddenly shifted inthe scene.

Accordingly in a variation of the foregoing embodiment of the invention,when the user has set the zoom position to be within the first opticalzoom range, the first imaging stage, including the first zoom lens andits associated image sensor, is used to set the initial focus for takingan image, such as a video image, while the second imaging stage,including the second zoom lens and its associated image sensor,simultaneously provides a continuous “whole-way” autofocus input imageto determine if the focus of the first zoom lens should be adjusted as aresult of subject motion. Since the second zoom lens and its associatedimage sensor are not used to capture the motion images, the focusdistance can be adjusted from the near focus to the infinity position,in order to determine the best focus setting without affecting thecaptured motion images. When the user adjusts the zoom position to beoutside the first zoom range, the camera automatically switches to thesecond imaging stage, using the second zoom lens and its associatedsensor to capture the motion sequence, and begins using the firstimaging stage, with its first lens and associated image sensor, tosimultaneously determine if the focus of the second zoom lens should beadjusted as a result of subject motion.

The two image capture stages may also be configured as a high resolutionrangefinder in order to determine the distances to different portions ofthe scene in the form of a range map. The range map is then used tomodify the captured image signal or the output image for a variety ofpurposes, such as (without limitation): to improve image processing andenable improved image quality; to improve object identification withinthe image; to enable object extraction from an image; to enable motiontracking of objects within multiple images; to enable reduced depth offield images by blurring of objects outside of the desired depth offield; to improve the dynamic range within images; to reduce exposureissues introduced by use of a flash; and to improve scene balance withinthe image.

The first imaging stage, including a first zoom lens and a first imagesensor, may also be used to capture a first (i.e., primary) still imageat a first (i.e., primary) focus distance, while a second imaging stage,including a second zoom lens and a second image sensor, is used tosimultaneously capture a second (i.e., secondary) still image at asecond (i.e., secondary) focus distance. The sensor output from thesecond imaging stage is used as a secondary output image for modifyingthe primary output image, thereby generating an enhanced primary imagesignal. For instance, the secondary still image is used to provide anenhancement signal that may, e.g., sharpen portions of the primary stillimage that are positioned near the secondary focus distance or maymodify the dynamic range of the primary still image.

As mentioned above, the images from both imaging stages may be used togenerate a range map identifying the distances to the different portionsof the scene. In another variation described herein, the camera furtherincludes a GPS unit for providing GPS coordinates for the location ofthe camera and an electronic compass for providing the pointingdirection of the camera. Thereupon, the GPS coordinates for the locationof the camera, the pointing direction of the camera, and distanceoffsets from the range map may be used in order to generate GPScoordinates for portions of the scene.

FIG. 1 depicts a block diagram of an image capture assembly 10Aaccording to the first embodiment of the present invention. Though notan essential aspect of the invention, the image capture assembly 10A ispreferably a portable battery operated device, small enough to be easilyhandheld by a user when capturing and reviewing images. In the preferredembodiment, the image capture assembly 10A is a digital camera thatproduces both still images and motion video images that are stored on aremovable memory card 54. Alternatively, the digital camera may produceand store only motion images or only still images.

The image capture assembly 10A includes two imaging stages 1 and 2, bothwith zoom lenses 3 and 4. (These stages will hereinafter be referred toin the specification as image capture stages, although in most casesonly one stage—at a time—is capturing an image that is stored on theremovable memory card 54.) The first zoom lens 3 is controlled by afirst lens focus adjuster, e.g., zoom and focus motors 5 a, and providesan image to a first image sensor 12. The second zoom lens 4 iscontrolled by a second lens focus adjuster, e.g., zoom and focus motors5 b, and provides an image to a second image sensor 14. An adjustableaperture and shutter assembly in each zoom lens (not shown) is used tocontrol the exposure to image sensors 12 and 14.

FIGS. 2A and 2B show perspective views of the image capture assembly 10Adescribed in relation to FIG. 1. FIG. 2A is a frontal view of the imagecapture assembly 10A, showing the first zoom lens 3, the second zoomlens 4 and a flash 48. FIG. 2B is a rear view of the camera 10A, showinga color LCD image display 70 and a number of user controls 42, includinga shutter button 42 a for enabling an image capture sequence, a zoombutton 42 c for enabling a selection of a zoom setting, and amulti-position selector 42 d for navigating through images, menu choicesand the like that are displayed on the color LCD image display 70.

The image capture stages 1 and 2 comprise the zoom lenses 3 and 4 andthe image sensors 12 and 14, as shown in FIG. 1. While zoom lenses 3 and4 are offset vertically, as shown in FIG. 2 a, the zoom lenses 3 and 4could be offset in other directions, such as horizontally. In addition,according to another embodiment of the invention, one (or both) of thezoom lenses 3 and 4 could be replaced with a fixed focal length lens. Inall cases, the optical axes of the zoom lenses 3 and 4 and the sensors12 and 14 are generally aligned with respect to each other so as to beviewing substantially the same scene, albeit typically with differentfields of view. The configuration of the optical components are furtherdescribed in the aforementioned, commonly assigned U.S. patentapplication Ser. No. 11/062,174, the disclosure of which is incorporatedherein by reference, which includes several embodiments in which morethan two image capture stages are used. The configuration of the zoomlenses 3 and 4 in the image capture stages 1 and 2 can include foldedoptical paths to change the overall dimensions of the image capturestages 1 and 2; however, folded optical paths are not necessary forpractice of the present invention.

In a preferred embodiment, the image sensors 12 and 14 are single-chipcolor megapixel CCD sensors, using the well-known Bayer color filterpattern in order to capture color images, although other sensors, suchas CMOS sensors, and other color filter arrays, such as stripe filters,may be used equally well without limitation according to the invention.The image sensors 12 and 14 may have a variety of aspect ratios, forexample, a 4:3 image aspect ratio and a variety of resolutions, forexample, a total of 6.1 MP effective megapixels (million pixels), with,in this particular case, 2848 active columns of pixels×2144 active rowsof pixels. It should also be understood that the image sensors 12 and 14do not have to have the same specifications. For instance, in someembodiments disclosed in the aforementioned, commonly assigned U.S.patent application Ser. No. 11/062,174, the size, resolution, colorfilter array, spectral sensitivity and aspect ratio of the image sensors12 and 14 can be different.

A control processor and timing generator 40 controls the first imagesensor 12 by supplying signals to clock drivers 13, and controls thesecond image sensor 14 by supplying signals to clock drivers 15. Thecontrol processor and timing generator 40 also controls the zoom andfocus motors 5 a and 5 b, an auto exposure detector 46, user controls42, first and second digital multiplexer control elements 34 and 36, andthe flash 48 for emitting light to illuminate the scene. The usercontrols 42 are used to control the operation of the digital camera 10A,as also described earlier in reference to FIG. 2B.

An analog output signal 12 e from the first image sensor 12 is amplifiedand converted to a first digital image signal by a first analog signalprocessor 22. The digitized first digital image signal is provided to afirst input of the first digital multiplexer control element 34 and to afirst input of the second digital multiplexer control element 36. Ananalog output signal 14 e from the second image sensor 14 is amplifiedand converted to a second digital image signal by a second analog signalprocessor 24. The digitized second digital image signal is provided to asecond input of the digital multiplexer control element 34 and a secondinput of a second digital multiplexer control element 36. The functionof the first multiplexer 34 is to select either the first sensor output12 e from the first image sensor 12, or the second sensor output 14 efrom the second image sensor 14 as the image capture signal. Thefunction of the second multiplexer 36 is to select either the secondsensor output 14e from the second image sensor 14 or the first sensoroutput 12 e from the first image sensor 12 as the autofocus imagesignal, which is provided to an image processor 50.

The control processor and timing generator 40 controls the digitalmultiplexers 34 and 36 in order to select one of the sensor outputs (12e or 14 e) as the captured image signal, and to select the other sensoroutput (14 e or 12 e) as the autofocus image signal. The digital dataprovided by the first digital multiplexer control element 34 istemporarily stored in a DRAM buffer memory 38 and subsequently processedby the image processor 50 to produce a processed digital image file,which may contain a still digital image or a video image. The digitaldata provided by the second digital multiplexer control element 36 isprovided to the image processor 50, which performs autofocuscalculations as will be described later in reference to FIGS. 4, 5, 6and 7.

Briefly summarized, the image processor 50 produces the focus detectionsignals that drive the first and second focus adjusters, that is, thezoom and focus motors 5 a and 5 b. The control processor and timinggenerator 40, in combination with the image processor 50, either (a)selects the sensor output 12 e from the first imaging stage 1 as thecaptured image signal and uses the sensor output 14 e from the secondimaging stage 2 to generate the focus detection signal for the selectedimaging stage 1 or (b) selects the sensor output 14 e from the secondimaging stage 2 as the captured image signal and uses the sensor output12 e from the first imaging stage 1 to generate the focus detectionsignal for the selected imaging stage 2. In such a manner, the focusdetection signal is applied to the zoom and focus motors 5 a and 5 b ofthe selected imaging stage in order to adjust the focus of the imageproviding the sensor output for the captured image signal.

The processing performed by the image processor 50 is controlled byfirmware stored in a firmware memory 58, which may be flash EPROM memoryor any other form of appropriate memory. The processor 50 processes thedigital input image from the DRAM buffer memory 38, using a RAM memory56 to store intermediate results during the processing stage. Theprocessed digital image file is provided to a memory card interface 52,which stores the digital image file on the removable memory card 54.Removable memory cards 54 are one type of removable digital imagestorage medium, and are available in several different physical formats.For example, the removable memory card 54 can include (withoutlimitation) memory cards adapted to well-known formats, such as theCompact Flash, SmartMedia, MemoryStick, MMC, SD, or XD memory cardformats. Other types of removable digital image storage media, such asmagnetic hard drives, magnetic tape, or optical disks, can alternativelybe used to store the still and motion digital images. Alternatively, thedigital camera 10A can use internal non-volatile memory (not shown),such as internal Flash EPROM memory to store the processed digital imagefiles. In such an embodiment, the memory card interface 52 and theremovable memory card 54 are not needed.

The image processor 50 also receives input from a global positioningsystem (GPS) unit 57, which enables the image processor 50 to determinethe GPS coordinates (i.e., location) of the camera at any appropriatetime, e.g., when an image is captured. The image processor also receivesdirectional input from an electronic compass 59, which enables the imageprocessor 50 to determine which direction the camera is pointed, e.g.,when an image is captured. The image processor 50 performs various otherimage processing functions, including color interpolation followed bycolor and tone correction, in order to produce rendered sRGB image data.The rendered sRGB image data is then JPEG compressed and stored as aJPEG image file on the removable memory card 54. The rendered sRGB imagedata may also be provided to a host PC 66 via a host interface 62communicating over a suitable interconnection 64, such as a SCSIconnection, a USB connection or a Firewire connection. The JPEG filepreferably uses the so-called “Exif” image format defined in “DigitalStill Camera Image File Format (Exif)” version 2.2 by the JapanElectronics and Information Technology Industries Association (JEITA),Tokyo, Japan. This format includes an Exif application segment thatstores particular image metadata, including the date/time the image wascaptured, as well as the lens f/number, GPS location and pointingdirection when the image was captured and other camera settings.

It should be noted that the image processor 50, while typically aprogrammable image processor, can alternatively be, for example, ahard-wired custom integrated circuit (IC) processor, a general purposemicroprocessor, or a combination of hard-wired custom IC andprogrammable processors. Furthermore, one or more of the functions shownas separate blocks in FIG. 1, such as the digital multiplexer controlelements 34 and 36, the DRAM buffer memory 38, and the RAM memory 58,can be incorporated in an IC containing the image processor 50. Itshould also be noted that the functions of at least certain portions ofthe control processor 40 and the image processor 50 may be merged asneeded for purposes of certain applications and discussions, such as inthe reference to a processing stage in this description and in certainof the claims, where, e.g., selection of a sensor output (as by thecontrol processor 40) and generation of a focus signal (as by the imageprocessor 50) are referred to. In other words, the recitation of aprocessing stage is intended to encompass the recited functions, whetherthey are to be found in one or more actual processing elements,circuits, or the like.

In a further embodiment of the present invention, the digital camera 10Ais included as part of a camera phone. In such an embodiment, the imageprocessor 50 also interfaces to a cellular processor 90, which uses acellular modem 92 to transmit digital images to a cellular network (notshown) using radio frequency transmissions via an antenna 94. In someembodiments of the present invention, the two image capture stages 1 and2, and the zoom and focus motors 5 a and 5 b may be part of anintegrated assembly. In addition, the clock drivers 13 and 15, as wellas the analog signal processors 22 and 24 and the analog/digitalconverters included therewith, may be part of the integrated assembly.

FIG. 3 depicts a flow diagram showing a method for capturing digitalimages using the image capture assembly of FIG. 1. In block 100, whenthe camera 10A is turned ON using a power switch (not shown), the zoomlenses 3 and 4 are set to their default positions, which is preferably awide angle position where the output of the first image sensor 12 isused to capture images in a preview mode for display on the color LCDimage display 70 to enable the user to compose the images to becaptured. As part of composing the images, in block 102 the usertypically presses the zoom button 42 c in order to set a desired fieldof view for the digital camera 10A.

In block 102, the zoom position setting is compared to a value X atwhich the image capture function switches from the first image capturestage to the second image capture stage. In block 104, if the zoomposition setting is less than X (a negative response to block 102), thenthe first image capture stage 1 is used to capture images in the previewmode, while the second image capture stage 2 is used to captureautofocus images. The first image capture stage 1 continues to captureimages for preview on the display 70 (block 110) while, in block 106,the second image capture stage 2 is used to capture autofocus images forautofocus of the first image capture stage 1, which are processed by theimage processor 50 and used in block 108 to focus the first imagecapture stage 1.

In block 112, if the zoom button 42 c is not pressed, and in block 114if the capture button is pressed, a digital image is captured in block116 with the first image capture stage 1. Alternatively, if the zoombutton is pressed in block 112, control is returned to block 102, and ifthe capture button is not pressed in block 114, control is returned toblock 106.

In block 124, if the zoom position setting is greater than X (a positiveresponse to block 102), then the second image capture stage 2 is used tocapture images in the preview mode, while the first image capture stage1 is used to capture autofocus images. The second image capture stage 2continues to capture images for preview on the display 70 (block 130)while, in block 126, the first image capture stage 1 is used to captureautofocus images for autofocus of the second image capture stage 2,which are processed by the image processor 50 to generate a focusdetection signal that is used in block 128 to focus the second imagecapture stage 2.

In block 132, if the zoom button 42 c is not pressed, and in block 134if the capture button is pressed, a digital image is captured in block136 with the second image capture stage 2. Alternatively, if the zoombutton is pressed in block 132, control is returned to block 102, and ifthe capture button is not pressed in block 134, control is returned toblock 126.

A flow chart of an autofocus process using the two image capture stagesshown in FIG. 3 in a rangefinder configuration is shown in FIG. 4, wherethe rangefinder method is used in blocks 108 and 128 of FIG. 3 toautofocus the images from the first and second image capture stages. Inblock 250, the user determines the zoom position by adjusting the zoomcontrol 42 c on the camera, which in turn dictates, as described above,which image capture stage will be used to capture the final image andwhich image capture stage will be used only for autofocus images (block252). The image capture stage that will not be used for capture of thefinal image is zoomed to a position closest to the transition zoomposition between the two zoom lens systems in the two image capturestages (block 254). The focus lenses of the two image stages are movedto their respective hyperfocal positions, wherein the greatest focusrange is produced (block 256). In block 258, the user presses thecapture button 42 a from a position S0 to a position S1 to initiate theautofocus sequence (and the autoexposure sequence). (The capture buttonhas 3 positions: S0 is the neutral position that the button maintainsprior to the operator touching the capture button; S1 is theintermediate position in which the camera performs autofocus andautoexposure; S2 is the final position in which the camera performs afinal autoexposure and captures the final image.)

Autofocus images are then captured by both image stages (block 260) withtheir zoom lenses at their respective zoom positions. The autofocusimage from the image stage in the lower zoom position, i.e., where thezoom position is less than X (see block 102 in FIG. 3), is then croppedand upsampled so that corresponding features in the two autofocus imagesspan the same number of pixels (block 262). The cropped and upsampledautofocus image is then correlated with the other autofocus image toidentify the pixel shift between the two autofocus images (block 264)and thereby produce the focus detection signal. FIG. 17A shows arepresentation of the autofocus image 350 as captured from the higherzoom position image stage. FIG. 17B shows a representation of a croppedand upsampled autofocus image 352 from the image stage in the lower zoomposition. These representations show the offset between the two imagesthat is used to determine the distance to portions of the scene. Inblock 266, a calibration factor is then applied to the focus detectionsignal to determine the distance that the focus lens must be moved toproduce a best focus condition for the image capture. In block 268, thefocus detection signal is applied to the zoom and focus motor 5 a andthe focus lens is then moved the determined distance in the imagecapture stage that will be used for the final image capture to producethe condition for best focus 268. When the user pushes the capturebutton from S1 to S2, the image capture stage capture the final image.

A flow chart of an autofocus process using the two image capture stagesshown in FIG. 3 in a well known “hill climb” contrast comparison methodis shown in FIG. 5, where the “hill climb” contrast comparison method isused in blocks 108 and 128 of FIG. 3 to autofocus the images from thefirst and second image capture stages. In block 250, the user selects azoom position. The zoom position determines which imaging stage will beused as the capture stage (block 252). Then, in block 254, the imagecapture stage not used for capture is zoomed to the point closest to theuser selected zoom position from block 250. In block 258, the userpushes the capture button 42 a from the S0 position to the S1 positionto initiate an autofocus sequence. Then, in block 272, both imagecapture stages are autofocused by the “hill climb” method. When the userpushes the capture button from the S1 position to the S2 position (block274), video images are continuously captured by the capture stage (block276).

In order to maintain focus with a minimal amount of hunting, focus iscontinually checked in block 278 with the image capture stage not beingused for capture using the “hill climb” contrast comparison method.Then, in the decision block 280, if the focus is good, control isreturned to block 276 and video images are continuously captured by thecapture stage. If, in the decision block 280, if the focus is not good,then the focus lens adjustment is identified (block 282) that is neededto produce best focus on the image capture stage not being used forcapture. In block 284, the autofocus “hill climb” method calibrationcurve and the identified focus lens adjustment is used to determine themovement of the focus lens needed in the capture stage to produce bestfocus, thereby producing a focus change detection signal. Finally, inblock 286, the focus change detection signal is applied to the zoom andfocus motor 5 a or 5 b and the focus lens in the capture stage is movedto the new best focus position, and control is returned to block 276 andvideo images are continuously captured by the capture stage.

Alternatively, and also in order to maintain focus with a minimal amountof hunting, focus is continually checked in block 278 with the imagecapture stage not being used for capture using the “whole way” autofocusmethod. Accordingly, in block 284, an autofocus “whole way” methodcalibration curve and the identified focus lens adjustment is used todetermine the movement of the focus lens needed in the capture stage toproduce best focus, thereby producing a focus change detection signal.Finally, in block 286, the focus change detection signal is applied tothe zoom and focus motor 5 a or 5 b and the focus lens in the capturestage is moved to the new best focus position, and control is returnedto block 276 and video images are continuously captured by the capturestage.

Calibration curves are used in both FIGS. 4 and 5 to determine themovement of the focus lens needed in the capture stage to produce bestfocus. FIGS. 6 and 7 depict flow diagrams for calculating these curves.More specifically, FIG. 6 depicts calculation of the autofocusrangefinder calibration curve used in block 266 of FIG. 4. In block 300of FIG. 6, a series of image sets are captured with objects at knowndistances, using the shorter focal length first image capture stage andthe longer focal length second image capture stage at a series of focuslens positions. Then, in block 302, the autofocus image from the lowerfocal length first image stage is cropped and upsampled so thatcorresponding features in the two autofocus images span the same numberof pixels as shown in FIGS. 17A and 17B. In block 304, the images fromthe second image capture stage are correlated to corresponding portionsof the images from the cropped and upsampled image from the first imagecapture stage to determine the pixel offset between the images in eachimage set. Thereupon, in block 306, the data of pixel offset betweenimages in each image set versus known distance to objects is stored asthe autofocus rangefinder calibration curve for use in block 266 of FIG.4.

FIG. 7 depicts calculation of the “hill climb” calibration curve used inblock 284 of FIG. 5. In block 400 of FIG. 7, a series of image sets arecaptured with objects at known distances, using the first image capturestage and the second image capture stage—wherein autofocus is done bythe “hill climb” method for each image. Then, in block 402, the focuslens positions is compared for the two image capture stages versus thedistance to the focused objects in the image sets. Then, the data offocus lens positions of the first image capture stage versus the focuslens positions of the second image capture stage for the same distanceto focused objects in the images is stored as an autofocus “hill climb”method calibration curve for use in block 284 of FIG. 5.

FIG. 8 depicts a flow diagram showing a method for capturing videoimages using the image capture assembly of FIG. 1. Much of the flowdiagram duplicates the functional elements shown in FIG. 3, and will notbe repeated here where the same functions and reference characters areindicated. If the zoom position setting is less than X (a negativeresponse to block 102), then the first image capture stage 1 is used tocapture video images, while the second image capture stage 2 is used tocapture autofocus images. The focus of the first image capture stage isperformed as described in FIG. 3, except it is now for a video image.Subsequently, in block 112 in FIG. 8, if the zoom button 42 c is notpressed, and in block 114 if the capture button is pressed, a videoimage is captured in block 118 by the first image capture stage 1. Thevideo image is checked for focus quality in block 119, and if there is aneed to refocus, control is returned to block 106 and the focus changedetection signal is generated, which is used in block 108 to drive thefocus motor 5 a for the first image capture stage 1. If there is no needto refocus, then control is returned to block 112.

If the zoom position setting is greater than X (a positive response toblock 102), then the second image capture stage 2 is used to capturevideo images, while the first image capture stage 1 is used to captureautofocus images. The focus of the second image capture stage isperformed as described in FIG. 3, except it is now for a video image.Subsequently, in block 132, if the zoom button 42 c is not pressed, andif in block 134 if the capture button is pressed, a video image iscaptured in block 138 with the second image capture stage 2. The videoimage is checked for focus quality in block 139, and if there is a needto refocus, control is returned to block 126 and the focus changedetection signal is generated, which is used in block 128 to drive thefocus motor 5 b for the second image capture stage 2. If there is noneed to refocus, then control is returned to block 132.

A flow chart of an autofocus process using the two image capture stagesshown in FIG. 8 in a rangefinder configuration is shown in FIG. 9, wherethe rangefinder method is used in blocks 108 and 128 of FIG. 8 toautofocus the images from the first and second image capture stages. Inblock 440 of FIG. 9, a first autofocus image is captured with the lowerfocal length image capture stage. Then, in block 442, the autofocusimage from the image stage in the lower focal length is cropped andupsampled so that corresponding features in the two autofocus imagesspan the same number of pixels. Meanwhile, in block 448, a secondautofocus image is captured with the longer focal length image capturestage. The second autofocus image is correlated in block 444 with thecropped and upsampled image to determine the pixel offset between theimages for different portions of the images. Then, in block 446, thefocus correction needed is determined from the offset and the autofocusrangefinder calibration curve (which was calculated in FIG. 6).

A flow chart of an autofocus process using the two image capture stagesshown in FIG. 8 in a “hill climb” contrast comparison method is shown inFIG. 10, where the “hill climb” contrast comparison method is used inblocks 108 and 128 of FIG. 8 to autofocus the images from the first andsecond image capture stages. In block 460 of FIG. 10, a first image iscaptured with the first image capture stage, wherein the autofocus isdone by the “hill climb” method. Then, in block 462, the image from thefirst image capture stage is provided as a preview image on the display.Meanwhile, in block 466, a second image is captured with the secondimage capture stage, wherein the autofocus is done by the “hill climb”method. Then, in block 468, another, subsequent image is captured withthe second image capture stage, wherein the autofocus is also done bythe “hill climb” method. In block 470, the focus conditions for thesecond image are compared to those of the subsequent image. Then, if thefocus conditions have changed (positive response to block 472), thefocus conditions are changed for the first image capture stage based onthe change of the focus conditions for the second image capture stageand the autofocus “hill climb” method calibration curve (which wascalculated in FIG. 7). If the focus conditions have not changed(negative response to block 472), control is returned to block 468 andanother image is captured with the second image capture stage.

Alternatively, and in order to maintain focus with a minimal amount ofhunting, focus is continually checked in block 470 with the second imagecapture stage using the “whole way” autofocus method. Accordingly, ifthe focus conditions have changed (positive response to block 472), thefocus conditions are changed for the first image capture stage based onthe change of the focus conditions for the second image capture stageand an autofocus “whole way” method calibration curve. If the focusconditions have not changed (negative response to block 472), control isreturned to block 468 and another image is captured with the secondimage capture stage.

FIG. 11 depicts a flow diagram showing a method for processing imagescaptured using the image capture assemblies of FIGS. 3 or 8, wherein arange map is produced. (Certain parts of the diagram bear the samefunctions and reference characters as used in FIG. 9.) Methods toproduce a rangemap are well known to those skilled in the art; forexample, a description of a method for producing a rangemap or depth mapfrom a disparity map produced from the pixel offset information for aset of images captured by multiple cameras with similar fields of viewis described in U.S. Patent Application Publication Number 2006/0193509(published Aug. 31, 2006 in the names of Antonio Criminisi et al., andentitled “Stereo-based Image Processing”), which is incorporated hereinby reference. In the case of the present patent application, two or moreimage capture devices are included in a single electronic camera. Sincethe two or more image capture devices have different focal lengths, atleast one of the images must be modified to make the two or more imagescomparable to enable the pixel offsets to be determined. Referring nowto FIG. 11, in block 440 a first autofocus image is captured with thelower focal length image capture stage and, in block 442, the autofocusimage from the image capture stage in the lower zoom position is croppedand upsampled so that corresponding features in the two autofocus imagesspan the same numbers of pixels. Meanwhile, in block 448, a secondautofocus image is captured with the higher focal length image capturestage. Then, in block 480, the second autofocus image is correlated withthe cropped and upsampled image to determine the pixel offset betweenthe images for different portions of the images. The pixel offsets arethen converted in block 482 to distances from the image capture deviceusing the autofocus rangefinder calibration curve. A map is thenproduced in block 484 showing the distances to different portions of theimages.

The distance from the image capture device to portions of the scene canbe calculated from the measured pixel offsets between correspondingportions of the first and second autofocus images, that is, betweencorresponding portions of the cropped and upsampled image obtained inblock 442 from the first autofocus image and the second autofocus imageobtained from block 448. The relationship between the pixel offset asexperienced on the image sensors p, the pixel size m, the spacingbetween lenses s, the effective focal length of the lens f and thedistance d to the portion of the scene is given as

d=s·f/(p·m)

Table 1 shows typical pixel offsets for an image capture device asdescribed. In a preferred embodiment, the relationship between pixeloffset and distance to portions of the scene is calibrated againstobjects in a scene with known distances to compensate for any unexpectedvariations in dimensions and any angular tilt between the two lensassemblies.

TABLE 1 Pixel size (mm) 0.002 Separation between lenses (mm) 20 FocalLength (mm) 6 Distance (ft) Distance (mm) Offset (pixels)   0.5 152.4393.7 1 304.8 196.9 2 609.6 98.4 4 1219.2 49.2 8 2438.4 24.6 16  4876.812.3 32  9753.6 6.2 64  19507.2 3.1 128  39014.4 1.5

As mentioned earlier, The range map is then used to modify the capturedimage signal or the output image for a variety of purposes, such as(without limitation):

-   -   a) to improve object identification within the image by        identifying the continuous boundaries of the object so the shape        of the object can be defined;    -   b) to enable object extraction from an image by identifying the        continuous boundaries of the object so it can be segmented        within the image;    -   c) to enable motion tracking of objects within multiple images        by identifying objects so they can be tracked as the same object        between images;    -   d) to enable dynamic depth of field images by blurring of        portions of the image that correspond to areas of the scene that        lie outside of the desired depth of field;    -   e) to improve the dynamic range within images by applying gain        adjustments to objects as a whole;    -   f) to reduce exposure issues introduced by use of a flash by        reducing the gain on the portions of the image that correspond        to objects in the foreground and increasing the gain on objects        in the background;    -   g) to improve scene balance within the image by enabling objects        in the foreground to be emphasized.

In order to understand the use of a range map for purposes such as notedabove, it is helpful to consider an example. Assume that auser/photographer has a great picture of the Alaskan mountains—beautifulclouded sky and white-capped mountains in the most distant ranges,flowers carpeting the fields in the mid range, and a black dog sittingin the foreground about 5 feet away. However, the clouds are blown out(over-exposed), as are the white-capped mountains. The black dog is toodark (underexposed) and out of focus (because, e.g., the camera was seton landscape mode). Using the range data, several features of the imagecan be modified. The exposure for various locations can be improved byapplying gain adjustments to selected object portions of the image: inparticular, e.g., to the cloud detail, the snow detail, and the fur onthe black dog. More generally, the range map can be used to improvedynamic range within the output image by applying gain adjustments toobjects as a whole within the output image, and independently of theirposition in the range map. Moreover, the depth of field can be adjustedso that, e.g., the dog is in focus, the mountains are in focus and soare those great flowers. Or, if the user really wants to emphasize thedog more than the beautiful scenery, the range data can be used toisolate the mountains and the flowers, which can then be blurred, andfurther to isolate the dog, which is sharpened to obtain a nice sharpimage. As can be understood, given the availability of a range mapaccording to the invention, there are numerous other uses that would beavailable for artistically optimizing the image. For instance, the usercan make a dynamic depth of field, that is, with mixed regions of theranges in focus. More generally, the range map can be used to enabledynamic depth of field images by blurring portions of the output image,independently of their position in the range map, that correspond toareas of the scene that lie outside of a desired depth of field for afeatured portion of the image. For example, the dog and mountains,albeit they are at opposite range extremes, could be brought in focusbecause they are the regions of interest and the flowers in the midrange can be blurred smoothly.

FIGS. 12 and 13 show block diagrams of digital cameras using differentvariations of the optical image capture stages. Most componentssubsequent to the image capture stages are the same as those describedin connection with FIG. 1, and will not be further described here. FIG.12 depicts a block diagram of a digital camera having a first imagecapture stage 71 and a second image capture stage 2. The first stage 71includes a first fixed focal length lens 73 and the first image sensor12, and the second stage 2 includes the second (zoom) lens 4 and secondimage sensor 14. The focusing according to the invention is carried outbetween the focus adjustments of the first and second image capturestages 71 and 2.

Because the focal length of the fixed focal length lens typicallygenerates an ultra-wide angle field of view, e.g., 22 mm equiv., it mayhave a fixed focus set to a distance near the lens hyperfocal distanceof, e.g., 8 feet, so that objects from 4 feet to infinity are in focus.Therefore, the fixed focal length lens does not need to include a focusadjustment. The fixed focal length lens includes an adjustable apertureand shutter assembly (not shown) to control the exposure of the imagesensor.

FIG. 13 depicts a block diagram of a digital camera having a first imagecapture stage 71, a second image capture stage 2, and a third imagecapture stage 74. The first stage 71 includes a first fixed focal lengthlens 73 with a first image sensor 12, the second stage 2 includes asecond (zoom) lens 4 with a second image sensor 14, and the third imagecapture stage 74 includes a third (zoom) lens 75 with a third imagesensor 16. According to this configuration, the first lens 73 istypically a ultra-wide angle lens, the second (zoom) lens 4 is typicallya wide angle zoom lens, and the third (zoom) lens 75 is typically atelephoto zoom lens. Since, as mentioned in the preceding paragraph, thefirst fixed focal length lens 73, being an ultra-wide angle lens, istypically not focused, the focusing according to the invention iscarried out between the second and third image capture stages 2 and 74.

FIG. 14 depicts a flow diagram showing a method for enhancing the depthof field of an image by using images from both image capture stages fromFIG. 1. In block 500 of FIG. 14, the zoom position is set to a defaultposition when the camera is powered on. In block 502, the zoom positionsetting is compared to a value X at which the image capture functionswitches from the first image capture stage to the second image capturestage. In block 504, if the zoom position setting is less than X (anegative response to block 502), then the first image capture stage 1 isused to capture images in the preview mode, while the second imagecapture stage 2 is used to capture autofocus images. The first imagecapture stage 1 continues to capture images for preview on the display70 (block 506) while the second image capture stage 2 is used to captureautofocus images for autofocus of the first image capture stage 1, whichare processed by the image processor 50 and used to focus the firstimage capture stage 1. Should the zoom button not be pressed (negativeresponse to block 508), and when the shutter button 42 a is pressed, aprimary still image is captured in block 510 using the first imagecapture stage set to a primary focus position. Then, in block 512, asecondary still image is captured using the second image capture stageset to a secondary focus position. Then, in block 514, the secondarystill image is used to enhance the depth of field of the primary image,for instance, where the secondary still image is used to provide anenhancement signal that can be used to sharpen portions of the primarystill image that are positioned near the secondary focus distance.

In block 524, if the zoom position setting is greater than X (a positiveresponse to block 502), then the second image capture stage 2 is used tocapture images in the preview mode, while the first image capture stage1 is used to capture autofocus images. The second image capture stage 2continues to capture images for preview on the display 70 (block 526)while the first image capture stage 1 is used to capture autofocusimages for autofocus of the second image capture stage 2, which areprocessed by the image processor 50 and used to focus the second imagecapture stage 2. Should the zoom button be pressed (positive response toblock 528), and when the shutter button 42 a is pressed, a primary stillimage is captured in block 530 using the second image capture stage setto a primary focus position. Then, in block 532, a secondary still imageis captured using the first image capture stage set to a secondary focusposition. Then, in block 534, the secondary still image is used toenhance the depth of field of the primary image, for instance, where thesecondary still image is used to sharpen portions of the primary stillimage that are positioned near the secondary focus distance.

As shown above, the enhancement signal is generated by the camera tosharpen portions of the primary still image that are positioned near thesecondary focus distance. However, the primary and secondary stillimages may submitted to an external processor, such as the host PC 66illustrated in FIG. 1, and the enhancement signal is generated by theexternal processor to sharpen portions of the primary still image thatare positioned near the secondary focus distance.

The concept of multiple lenses and multiple sensors, and the use of anintegrated image capture assembly, may be adapted for use in a cellphone of the type having a picture taking capability. Accordingly, andas shown in FIG. 15A, a cell phone 600 includes a phone stage comprisinga microphone 602 for capturing the voice of a caller, relatedelectronics (not shown) for processing the voice signals of the callerand the person called, and a speaker 604 for reproducing the voice ofthe one called. A keypad 606 is provided for entering phone numbers andimage capture commands, and a (LCD) display 608 for showingphone-related data and for reproducing images captured by the phone orreceived over the cellular network. The rear view of the cell phone 600shown in FIG. 15B identifies some of the internal components, includinga cellular image capture assembly 610 connected via the image processor50 (as shown in FIG. 1) to a cellular processing stage comprising thecellular processor 90 and the modem 92. The cellular processor 90receives and processes the image data from the image processor 50 andthe voice data captured by the microphone 602, and transfers the imageand voice data to the cellular modem 92. The cellular modem 92 convertsthe digital image and voice data into the appropriate format fortransmission by the antenna 94 to a cellular network.

As the cellular image capture assembly 610 is shown in FIGS. 16A and16B, where FIG. 16A is a top view of the assembly 610 taken along thelines 24B-24B in FIG. 16B, the assembly 610 comprises an integratedpackaging of the optical and imaging components on a common substrate620. More specifically, the assembly 610 includes a first fixed focallength lens 612 and a first image sensor 614, and a second fixed focallength lens 616 and a second image sensor 618. The first lens 612,preferably a fixed focal length wide angle lens (such as a 40 mm equiv.lens), forms an image on the first image sensor 614, and the second lens616, preferably a fixed focal length telephoto lens (such as 100 mmequiv. lens), forms an image on the second image sensor 618. Both of thelenses are oriented in the same direction in order to form images of thesame portion of the overall scene in front of them, albeit withdifferent fields of view.

Each lens 612 and 616 and each associated image sensor 614 and 618 aremounted to the substrate 620 with an IR cut filter 622 in between toreduce the incidence of IR radiation on the image pixels. Electroniccomponents 624, such as resistors, capacitors and power managementcomponents, are also mounted on the substrate 620. The image signals aretaken from the substrate 620 via a flex connector 626. The data takenfrom the assembly 610 may be raw image data, or if suitable processors(not shown) are on board the substrate 620, the data could be YUV imagedata or JPEG image data. Moreover, the image processor 50 may providedigital zooming between the wide angle and the telephoto focal lengths;the user may initiate such zooming via a user interface displayed on the(LCD) display 608 and by keying appropriate buttons on the keypad 606.Furthermore, the wide angle image sensor 614 may have high resolution,e.g., higher than that of the telephoto image sensor 618, in order toprovide a higher quality source image for the digital zooming.

In an embodiment according to the present invention, where both lenses612 and 616 are adjustable focus lenses, the image processor 50 either(a) selects the sensor output from the wide angle lens 612 as thecaptured image signal and uses the sensor output from the telephoto lens616 to generate a focus detection signal for the wide angle lens 612 or(b) selects the sensor output from the telephoto lens 616 as thecaptured image signal and uses the sensor output from the wide anglelens 612 to generate the focus detection signal for the telephoto lens616. The focus detection signal is then applied to the autofocussubsystem 628 of the telephoto lens 616 in order to adjust the focus ofthe image providing the sensor output for the captured image signal. Inthis embodiment, the wide angle lens 612 could instead be a zoom lens,such as a wide angle to normal angle zoom lens.

In another embodiment, the wide angle lens 612 is set to its hyperfocaldistance, which means it is in focus from a few feet to infinity withoutneed for any focus adjustment by the user. The telephoto lens 616 isautomatically focused by an auto focus subsystem 628. This is requiredbecause the hyperfocal distance increases as the focal length increases,and so the focus needs to be adjusted in order to obtain proper focusfor objects at typical (e.g. 4′ to 12′) distances. By using only onefocusing subsystem 628 for the telephoto lens 616, the cost and size canbe reduced.

An important constraint in this embodiment is the “z” dimension 630,which must be held to a very small figure consistent with a cell phonelayout and architecture. This may be obtained by careful choice of thetelephoto focal length and the size of the sensor. For example, the sizeof the sensor 616, and consequently the size of the image that must beproduced to fill the sensor, may be made small enough to reduce thefocal length to an acceptable z dimension 630.

In a further embodiment, the two lenses may have approximately identicalfocal lengths, with the imaging arrays being of different sizes. Withthe differently sized imaging arrays, each lens is designed to fill thearea of the imaging array and each lens-array combination will havesubstantially the same actual focal length, i.e., the same lens to arraydistance. However, the 35 mm equiv. of each lens will be different;consequently, each lens will have a different field of view.

While not shown in detail in FIGS. 16A and 16B, but similarly as wasexplained in connection with FIG. 1, an analog output signal from thefirst image sensor 614 is amplified by a first analog signal processorand provided to a first input of a control element, e.g., an analogmultiplexer control element provided as one of the electronic components624 on the substrate 620. The analog output signal from the second imagesensor 618 is amplified by a second analog signal processor and providedto a second input of the control element. The function of the controlelement is to select either the first sensor output from the first imagesensor 614 or the second sensor output from the second image sensor 618,depending on user input from the keypad 606 as to zoom selection,thereby providing a selected sensor output from the cellular imagecapture assembly 600 to the image processor 50.

In using the range map (generated according to FIG. 11), the GPSlocation and the pointing direction of the camera, which is provided bythe GPS unit 57 and the electronic compass 59 in the camera, arecombined with distances and directions from the camera to portions ofthe scene to determine the GPS locations for portions of the scene. FIG.18 depicts a flow chart for a method to determine GPS locations forportions of the scene. In block 750, the GPS location of the camera isdetermined by the GPS unit 57 in the camera. The pointing direction ofthe camera is determined in block 752 by the electronic compass 59 inthe camera. Distance offsets from the camera to portions of the sceneare provided by the range map, which is obtained in block 754 from aprocedure such as depicted in FIG. 11. Angular offsets from the pointingdirection of the camera (provided by the electronic compass 59) aredetermined in block 756 from the location of a portion of the scene inthe field of view in the image. GPS locations for portions of the sceneare determined in block 758 by adding the distance offsets and theangular offsets to the GPS location and pointing direction of thecamera. The GPS locations for portions of the scene are then stored inblock 760 in the metadata for the image or displayed as labels on theimage in the form of a GPS location map. Alternately, the GPS locationfor a portion of the image can be displayed on the electronic camera.Those skilled in the art will recognize that GPS locations of portionsof the scene can be used for a variety of purposes, including withoutlimitation: determining the identity of an object; providingnavigational directions to a location in a scene; identifying the fieldof view of a camera relative to a map or other geographical database,etc.

FIG. 19 depicts a flow diagram illustrating a process for selecting oneof the imaging stages in a dual lens camera system as the primarycapture unit, while relegating the other imaging stage to certain otherfunctions, such as scene analysis. More specifically, the power to thecamera is turned on and the initialization process begins (block 1100).After the initialization process is completed, the first and secondimaging stages 1 and 2 are set to their default zoom positions (block1102), which are predetermined initial zoom positions that determine theimages that are initially captured and displayed. Thereupon, the firstand second imaging stages capture and display first and second previewimages (block 1104) on the image display 70. These images could bedisplayed in several ways, for instance, side by side on the display asthumbnails or the like, one within the other, sequentially, etc. Next,the camera operator is asked to decide whether the first or secondpreview image should be the primary image (block 1106), where theprimary image will be the one that is captured and stored. The operatoris given a display time interval x in which to make this decision (block1110). If the shutter button is pressed during this interval (block1108), or if the operator fails to make a decision during this interval(yes to block 1110), a predetermined one of the imaging stages (thedefault imaging stage) is automatically set to be the primary captureunit (block 1112). (The default imaging stage could be pre-selected fora variety of reasons, one being to provide a comprehensive, wider angleview of the scene as the initial image.) Otherwise, if the operatorpicks one of the stages (yes to block 1106), the selected imaging stageis set to be the primary capture unit (block 1124). For eithersituation, the other (non-selected) imaging stage is designated as thescene analysis capture unit (block 1114). Thereupon, the primary captureunit is operated in the preview mode, as subsequently explained inconnection with FIGS. 20-22.

FIGS. 20-22 and 24-26 depict flow diagrams illustrating the usage oranalysis of the image from one imaging stage in a dual (or more) lenscamera system to modify the image produced by another imaging stage,that is, to influence, analyze, augment or otherwise change the imageproduced by the other imaging stage. For instance, in FIG. 20, once theprimary capture unit is put in the preview mode (as shown in FIG. 19),the scene analysis capture unit analyzes the scene (block 1200) and theimage processor 50 sets the primary capture unit parameters utilizingthe scene analysis data obtained by the scene analysis capture unit(block 1202). Such scene analysis data could include without limitationexposure data, dynamic range data, depth of field data, color balance,identification of different aspects of the scene including faces, grass,sunset, snow, etc, and the capture unit parameters could include withoutlimitation aperture value, exposure time, focus position, white balance,ISO setting, etc. The preview image is then captured by the primarycapture unit and displayed on the display 70 (block 1204). The sceneanalysis capture unit then continues to analyze the scene (block 1206),and if the scene conditions have not changed (no to block 1208) theprocess loops back to block 1204 and a preview image is again capturedby the primary capture unit and displayed on the display 70 (withoutchanging the capture parameters). If the scene conditions have changed(yes to block 1208), the process loops back to block 1202 and the imageprocessor 50 resets the primary capture unit parameters utilizing thescene analysis data obtained by the scene analysis capture unit, and thesubsequent process is repeated as before.

In FIG. 21, the primary capture unit parameters are only changed if thechange in scene conditions exceed a threshold value x. Thus, inutilizing the preview mode with a threshold, the scene analysis captureunit captures an image and analyzes the scene (block 1300). Then, theimage processor 50 sets the primary capture unit parameters utilizingthe scene analysis data obtained by the scene analysis capture unit(block 1302). A preview image is then captured by the primary captureunit and displayed on the display 70 (block 1304). Then, another imageis captured from the scene analysis capture unit (block 1306). If thescene conditions of this later image fail to change by a threshold equalto x (no to block 1308), that is, the scene conditions are considered tobe stable from image to image, the process loops back to block 1304 andanother preview image is captured by the primary capture unit anddisplayed on the display 70 (without changing the capture parameters).Then, another image is captured from the scene analysis capture unit(block 1306). Otherwise, when the scene condition change is greater thanthe threshold x and the situation is considered to be unstable (yes toblock 1308), the process loops back to block 1302 and the imageprocessor 50 resets the primary capture unit parameters utilizing thescene analysis data obtained by the scene analysis capture unit. Then,the process is repeated as before.

In FIG. 22, the consideration of the threshold scene conditions isenhanced by considering the relative zoom positions of the capture unitsand utilizing scene information from the images captured by both of thecapture units. In utilizing this enhanced preview mode, the zoomposition of the scene analysis capture unit is set relative to the zoomposition of the primary capture unit (block 1400). Then, the sceneanalysis capture unit captures an image (block 1402) and the imageprocessor 50 sets the primary capture unit parameters utilizing thescene analysis data obtained by the scene analysis capture unit (block1404). A preview image is then captured by the primary capture unit(block 1406) and the scene is analyzed utilizing the captured previewimage and the scene analysis data (block 1408). Then, the imageprocessor 50 sets the primary capture unit parameters utilizing theresults of the preceding scene analysis (block 1410), that is, resultsobtained by analyzing images from both capture units. Next, the previewimage is captured by the primary capture unit and displayed on thedisplay 70 (block 1412) and another image is captured from the sceneanalysis capture unit (block 1414). The scene is then analyzed byutilizing the captured preview image data and the scene analysis data(block 1416). If the scene conditions fail to change by a thresholdequal to x (no to block 1418), that is, the scene conditions areconsidered to be stable, the process loops back to block 1412 andanother preview image and scene analysis image are captured by theprimary capture unit and the scene analysis capture unit, respectively(blocks 1412 and 1414) and the aforementioned process continues.Otherwise, where the scene condition change is greater than thethreshold (yes to block 1400) and the situation is considered to beunstable, the process loops back to block 1410 and the image processor50 resets the primary capture unit parameters utilizing the results ofnew scene analysis data (obtained by the scene analysis capture unit inblock 1416), and the process is repeated.

During the operations shown in the preceding FIGS. 19-22, the previewprocess is continued as shown unless either the shutter button 42 a orthe zoom button 42 c is pressed. As shown in FIG. 23, if the zoom button42 c is pressed (block 1500), and if the requested zoom position is notwithin the zoom range of the primary capture unit (yes to block 1502),the functions of the capture units are reversed, that is, the currentscene analysis and primary capture units are reset to be the primarycapture unit and scene analysis capture unit, respectively. Then, thezoom position of the primary capture unit is set to the selected zoomposition (block 1506) and the process returns to the preview mode, asillustrated by FIGS. 20-23. However, if the zoom button 42 c is pressed(yes to block 1500), and if the requested zoom position is within thezoom range of the primary capture unit (no to block 1502), the functionsof the respective capture units are maintained the same as before andthe zoom position of the primary capture unit is set to the selectedzoom position (block 1506), and the process returns to the preview mode.

FIGS. 24-26 depict flow diagrams illustrating the usage or analysis ofthe image from one imaging stage in a dual (or more) lens camera systemto modify the image produced by another imaging stage, that is, toinfluence, analyze, augment or otherwise change the image produced bythe other imaging stage, specifically during the capture process.Referring to FIG. 24, to enter the capture mode (block 1600) accordingto a first capture embodiment, the shutter button 42 a is pressedhalf-way (S1)—otherwise, the process returns to the preview mode. In thecapture mode, a preview image is captured from the primary capture unit(block 1602). The scene is next analyzed utilizing the captured previewimage (block 1604). When the analysis is complete (yes to block 1606),the image processor 50 sets the primary capture unit parametersutilizing the results of the scene analysis (block 1608). Then, usingthe set parameters, a preview image is captured and displayed from theprimary capture unit (block 1610). When the shutter button 42 a is fullypressed (S2) so as to initiate image capture (yes to block 1612), aprimary image is captured from the primary capture unit using the setparameters (block 1614), and the process returns to the preview mode. Ifthe shutter button 42 a was not fully depressed (no to block 1612), thecamera checks to see if the focus or exposure lock is set (block 1616).If it is (yes to block 1616), the process loops back to block 1610 andanother preview image is captured and displayed from the primary captureunit, the condition of the shutter button 42 a is again checked (block1612), and the process continues as before. If the focus or exposurelock is not set (no to block 1616), the process loops back to the block1602 and a preview image is captured from the primary capture unit andthe scene is analyzed utilizing the captured preview image (block 1604).Then, the process continues as before.

Referring to FIG. 25, to enter the capture mode (block 1700) accordingto a second capture embodiment, the shutter button 42 a is pressed (S1)half-way. Then, a preview image is captured from the primary captureunit (block 1702). Next, an image is captured by the scene analysiscapture unit (block 1704), and the scene is analyzed utilizing both thecaptured preview image and the captured scene analysis image (block1706). The image processor 50 next sets the primary capture unitparameters utilizing the results of the combined scene analysis (block1708). Then, using the set parameters, a preview image is captured anddisplayed from the primary capture unit (block 1710). Next, and as shownbefore in FIG. 24, when the shutter button 42 a is fully (S2) pressed(yes to block 1712), a primary image is captured from the primarycapture unit using the set parameters (block 1714), and the processreturns to the preview mode. If the shutter button 42 a was not fullydepressed, the camera checks to see if the focus or exposure lock is set(block 1716). If it is, the process loops back to block 1710 and anotherpreview image is captured and displayed from the primary capture unit(block 1710), the condition of the shutter button 42 a is checked (block1712), and the process continues as before. If the focus or exposurelock is not set (no to block 1716), the process loops back to block1702, and a preview image is captured from the primary capture unit(block 1702) and the scene is analyzed utilizing the captured previewimage (block 1704). Then, the process continues as before.

Referring to FIG. 26, to enter the capture mode according to a thirdcapture embodiment, the shutter button 42 a is pressed (S1) half-way(block 1800). Then, a preview image is captured from the primary captureunit (block 1802) and an image is captured from the scene analysiscapture unit (block 1804). The scene is next analyzed utilizing both thecaptured preview image and the captured scene analysis image (block1806), and the image processor 50 sets the primary capture unitparameters utilizing the results of the combined scene analysis (block1808). The scene analysis capture unit is next designated as a secondarycapture unit (block 1810), and the image processor 50 sets the secondarycapture unit parameters utilizing the results of the combined sceneanalysis (block 1812). Next, when the shutter button 42 a is fully (S2)pressed (yes to block 1814), a primary image is captured from theprimary capture unit using the set parameters (block 1816) and anaugmentation image is captured from the scene analysis capture unit(block 1818), which is now designated the secondary capture unit. Thenan enhanced image is produced by the image processor 50 from the primaryand augmentation images (block 1820), and the process returns to thepreview mode. If the shutter button 42 a was not fully pressed (block1814), the process loops back to block 1802 and the process is resumedby again capturing images from the primary capture unit and the sceneanalysis capture unit (blocks 1802 and 1804).

Different types of augmentation or modification are contemplated inrelation to FIG. 26. In a first type of augmentation or modification,and as was depicted in connection with FIG. 14, an image is capturedfrom the primary capture unit at one focus position and another image iscaptured from the scene analysis capture unit (the secondary imagecapture unit) at another focus position. Then, the two images arecombined into a modified image with a broadened depth of field. Theadvantage is that this can be done without having to stop down theaperture of the primary lens to obtain the greater depth of field, whichis particularly useful for low light capture where a large aperture ispreferred.

In another type of augmentation or modification, the image could beexamined by the image processor 50 for its dynamic range. When thedynamic range exceeds a predetermined threshold, the scene analysiscapture unit is used to produce a secondary image with different dynamicrange than the primary image. For example, the primary image could becaptured at a normal exposure, while the secondary image could becaptured at more of an extreme exposure, i.e., either under- orover-exposed. Then, if for example, highlights are blown out in theprimary exposure, the secondary exposure could be underexposed tocapture detail in the highlights. Alternatively, if shadows are darkenedin the primary exposure, the secondary exposure could be overexposed tocapture detail in the shadows. Then, a modified image is created with abroadened dynamic range by replacing portions of the primary image withportions of the secondary image.

It should be understood that the images captured by the primary andsecondary capture units could be a still image or a video image, and inthe case of a video image could be a series of images. In either case,the still image or the series of images constituting the video signalmay be modified in accordance with the particular form of augmentationdescribed. For instance, an electronic camera incorporating the captureunits may produce a video image signal and the secondary output image isused to modify at least, e.g., the depth of field and/or the dynamicrange of the series of images constituting the video image signal.

The flow diagram shown in FIG. 14 can be modified to obtain a primaryimage that is enhanced specifically for dynamic range. For instance,referring to FIG. 14, when the shutter button 42 a is pressed, a primarystill image is captured (similar to blocks 510 or 530) using the first(or second) image capture stage set to a primary exposure level. Then,(similar to blocks 512 or 532) a secondary still image is captured usingthe second (or first image) capture stage set to a secondary exposurelevel. Then, (similar to blocks 514 or 534) the secondary still image isused to enhance the dynamic range of the primary image, for instance,where the secondary exposure, which is underexposed to capture detail inthe highlights, is combined with the primary image to obtain an extendeddynamic range primary image.

In a preferred method for producing the modified image, a sliding scaleis used to create the modified image in which the pixel values aredetermined by considering the pixel values of both the primary andsecondary images, as described in commonly-assigned, copending U.S.patent application Ser. No. 11/460,364 (which was filed Jul. 27, 2006 inthe names of John Border and Efrain Morales, and entitled “Producing anExtended Dynamic Range Digital Image”), which is incorporated herein byreference. It should be noted that having two exposure levels is toenable a correction for dynamic range is particularly important whenusing a flash, wherein overexposure conditions are often produced.Consequently, the exposure time of one of the capture stages will be setto a very short exposure or the timing of the exposure will be shiftedto be either just before the flash or just after the flash.

The augmentation process can also be applied in connection with imagepairs having different resolutions. For instance, in commonly-assigned,copending U.S. patent application Ser. No. 11/461,574 (which was filedAug. 1, 2006 in the names of John Border, Scott Cahall and JohnGriffith, and entitled “Producing Digital Image with DifferentResolution Portions”), which is incorporated herein by reference, afirst wide angle digital image of a scene and a second telephoto digitalimage of a portion of substantially the same scene are captured by twocapture stages. A composite image is then formed by combining a portionof the first wide angle digital image and a portion of the telephotodigital image, to produce a digital image with improved resolutionduring digital zooming. More specifically, digital zooming of thecomposite image produces a zoomed image with high resolution throughoutthe zoom range with improved image quality.

In a further embodiment, the primary capture stage and secondary capturestage are set for different exposure times so that different levels ofnoise and motion blur are present in the respective images. For example:the primary capture stage is set for a relatively long exposure so thatthe digital noise in the image is low, but any motion present eitherfrom movement of the camera or from movement of objects in the sceneresults in motion blur. Simultaneously, the secondary capture stage isset for a relatively fast exposure so that digital noise in the image ishigher, but the motion blur is less. The primary image and the secondaryimage are then compared to one another to identify motion blur. The gainof the secondary image is increased so the average pixel values in thesecondary image match those of the primary image. A modified image isthen created by replacing portions of the primary image with portions ofthe secondary image. In effect, portions of the primary image (areas oflower noise but with some motion blur) are replaced with correspondingportions of the secondary image (areas of higher noise but little or nomotion blur) to obtain a modified image with relatively low noise andgood sharpness.

The invention has been described in detail with particular reference tocertain preferred embodiments thereof, but it will be understood thatvariations and modifications can be effected within the spirit and scopeof the invention.

PARTS LIST

-   1 first imaging stage-   2 second imaging stage-   3 first zoom lens-   4 second zoom lens-   5 a zoom and focus motors-   5 b zoom and focus motors-   10A digital camera-   12 first image sensor-   12 e first sensor output-   13 clock drivers-   14 second image sensor-   14 e second image output-   15 clock drivers-   16 third image sensor-   17 clock drivers-   22 first analog signal processor (ASP1) & A/D converter-   24 second analog signal processor (ASP2) & A/D converter-   25 third analog signal processor (ASP2) & A/D converter-   34 first digital multiplexer control element-   36 second digital multiplexer control element-   37 third digital multiplexer control element-   38 DRAM buffer memory-   40 control processor and timing generator-   42 user controls-   42 a shutter button-   42 c zoom button-   42 d multi-position selector-   46 automatic exposure detector-   48 electronic flash-   50 image processor-   52 memory card interface-   54 removable memory card-   56 RAM memory-   57 GPS unit-   58 firmware memory-   59 electronic compass-   62 host interface-   64 interconnection-   66 host PC-   70 color LCD image display-   71 first image capture stage-   73 first fixed focal length lens-   74 third image capture stage-   75 third (zoom) lens-   90 cellular processor-   92 cellular modem-   94 antenna-   100 zoom lens setting block-   101 input zoom position block-   102 zoom position check block-   104 capture stage setting block-   106 second stage autofocus image capture block-   108 first stage focus block-   110 first stage preview capture and display block-   112 zoom button check block-   114 capture button check block-   116 first stage capture block-   118 first stage video capture block-   119 focus quality check block-   120 refocus check block-   124 capture stage setting block-   126 first stage autofocus image capture block-   128 second stage focus block-   130 second stage preview capture and display block-   132 zoom button check block-   134 capture button check block-   136 second stage capture block-   138 second stage video capture block-   139 focus quality check block-   140 refocus check block-   151 object-   152 first lens-   153 second lens-   154 first path-   155 second path-   156 focal plane-   157 first image-   158 second image-   170 first reference position-   171 first shifted position-   180 second reference position-   181 second shifted position-   190 first sensor array-   191 second sensor array-   197 low frequency bandpass filter loading stage-   198 comparison stage-   199 high frequency bandpass filter loading stage-   250 zoom position selection block-   252 image capture stage determination block-   254 non-capture stage zoom block-   256 hyperfocal setting block-   258 initiate autofocus sequence block-   260 capture autofocus images block-   262 crop and upsample block-   264 image correlation block-   266 best focus determination block-   268 focus lens movement block-   270 capture image block-   272 “hill climb” autofocus block-   274 capture button actuation block-   276 capture video block-   278 continuous “hill climb” focus check block-   280 focus check block-   282 best focus determination block-   284 focus lens movement block-   286 capture stage refocus block-   300 image sets capture block-   302 crop and upsample block-   304 correlate images block-   306 autofocus rangefinder calibration curve generation block-   350 representation of an autofocus image-   352 representation of a zoomed and cropped autofocus image-   400 image sets capture block-   402 position vs. distance comparison block-   404 autofocus “hill climb” calibration curve generation block-   440 lower focal length capture block-   442 crop and upsample lower focal length image block-   444 correlate images block-   446 determine focus correction block-   448 longer focal length capture block-   460 first stage “hill climb” first image capture block-   462 first stage preview image block-   464 refocus first stage block-   466 second stage “hill climb” second image capture block-   468 second stage another image capture block-   470 second stage focus conditions comparison block-   472 focus change check block-   480 correlate images block, determining pixel offsets-   482 pixel offset conversion block, using rangefinder calibration    curve-   484 range map production block-   500 default zoom position block-   502 zoom position check block-   504 capture stage setting block-   506 second stage preview capture, display and autofocus block-   508 zoom button check block-   510 first stage primary image capture at primary focus block-   512 second stage secondary image capture at secondary focus block-   514 primary image enhancement block, using secondary image-   524 capture stage setting block-   526 first stage preview capture, display and autofocus block-   528 zoom button check block-   530 second stage primary image capture at primary focus block-   532 first stage secondary image capture at secondary focus block-   534 primary image enhancement block, using secondary image-   600 cell phone-   602 microphone-   604 speaker-   606 keypad-   608 (LCD) display-   610 cellular image capture assembly-   612 first fixed focal length lens-   614 first image sensor-   616 second fixed focal length lens-   618 second image sensor-   620 substrate-   622 IR cut filter-   624 electronic components-   626 flex connector-   628 auto focus subsystem-   630 z dimension-   750 camera GPS location block-   752 camera pointing direction block-   754 distance offsets block-   756 angular offsets block-   758 scene GPS locations block-   760 GPS location storage block-   1100 power on block-   1102 default setting block-   1104 capture and display block-   1106 preview selection block-   1108 shutter check block-   1110 display time check block-   1112 set default capture unit block-   1114 set scene analysis capture unit block-   1124 set selected primary capture unit block-   1200 scene analysis block-   1202 set capture parameters block-   1204 capture and display preview image block-   1206 scene analysis block-   1208 scene condition check block-   1300 capture scene analysis image block-   1302 set capture parameters block-   1304 capture and display preview image block-   1306 scene analysis block-   1308 scene condition threshold check block-   1400 zoom position setting block-   1402 capture scene analysis image block-   1404 set capture parameters block-   1406 capture preview image block-   1408 scene analysis block, using preview and scene analysis data-   1410 set primary capture parameters block-   1412 capture and display preview image block-   1414 capture scene analysis image block-   1416 scene analysis block, using preview and scene analysis data-   1418 scene condition threshold check block-   1500 zoom button check block-   1502 zoom position check block-   1504 reverse capture unit assignment block-   1506 set primary capture unit zoom position block-   1600 shutter button check block-   1602 capture preview image block-   1604 scene analysis using preview image block-   1606 analysis complete check block-   1608 set primary capture parameters block-   1610 capture and display preview image block-   1612 shutter button check block-   1614 capture primary image block-   1616 focus/exposure lock check block-   1700 hutter button check block-   1702 capture preview image block-   1704 capture scene analysis image block-   1706 scene analysis using preview and scene image block-   1708 set primary capture parameters block-   1710 capture and display preview image block-   1712 shutter button check block-   1714 capture primary image block-   1716 focus/exposure lock check block-   1800 shutter button check block-   1802 capture preview image block-   1804 capture scene analysis image block-   1806 scene analysis using preview and scene image block-   1808 set primary capture parameters block-   1810 set scene analysis capture unit as secondary capture unit block-   1812 set secondary capture parameters block-   1814 shutter button check block-   1816 capture primary image block-   1818 capture augmentation image block-   1820 produce enhanced image block

1. An electronic camera for producing an output image of a scene from a captured image signal, said electronic camera comprising: a first imaging stage comprising a first image sensor for generating a first sensor output, a first lens for forming a first image of the scene on the first image sensor, and a first lens focus adjuster for adjusting focus of the first lens responsive to a first focus detection signal; a second imaging stage comprising a second image sensor for generating a second sensor output, a second lens for forming a second image of the scene on the second image sensor, and a second lens focus adjuster for adjusting focus of the second lens responsive to a second focus detection signal; and a processing stage for either (a) selecting the sensor output from the first imaging stage as the captured image signal and using the sensor output from the second imaging stage to generate the first focus detection signal for the selected imaging stage or (b) selecting the sensor output from the second imaging stage as the captured image signal and using the sensor output from the first imaging stage to generate the second focus detection signal for the selected imaging stage, whereby the focus detection signal is applied to the lens focus adjuster of the selected imaging stage in order to adjust the focus of the image providing the sensor output for the captured image signal.
 2. The electronic camera as claimed in claim 1 further comprising a user control that allows a user to select a focal length and wherein the processing stage is responsive to the user control for selecting one of the sensor outputs to provide the focus detection signal and the other of the sensor outputs to provide the captured image signal.
 3. The electronic camera as claimed in claim 1 wherein the focus detection signal is obtained by one of the image capture stages using a “hill climb” contrast comparison method.
 4. The electronic camera as claimed in claim 1 wherein the focus detection signal is obtained by one of the image capture stages using a continuous “whole way” auto focus method.
 5. The electronic camera as claimed in claim 1 wherein the lens focus adjusters continually readjust focus of the lenses responsive to a focus change detection signal, and wherein the sensor output from the imaging stage not providing the captured image signal is used to detect a change in focus of the image of the scene and to generate the focus change detection signal for the imaging stage providing the captured image signal, whereby the focus change detection signal is applied to the lens focus adjuster of the imaging stage providing the captured image signal in order to continually readjust the focus of the image.
 6. The electronic camera as claimed in claim 1 wherein the processing stage further uses the sensor outputs from both the first and second imaging stages in a rangefinder configuration to detect focus of the image of the scene and to generate the focus detection signal for the selected imaging stage.
 7. The electronic camera as claimed in claim 1 wherein the captured image signal is a still image signal.
 8. The electronic camera as claimed in claim 1 wherein the captured image signal is a video image signal.
 9. The electronic camera as claimed in claim 1 wherein at least one of the first lens and the second lens is a zoom lens.
 10. A video camera for producing an output video image of a scene from a captured image signal, said video camera comprising: a first imaging stage comprising a first image sensor for generating a first sensor output, a first lens for forming a first image of the scene on the first image sensor, and a first lens focus adjuster for continually readjusting focus of the first lens responsive to a focus change detection signal; a second imaging stage comprising a second image sensor for generating a second sensor output and a second lens for forming a second image of the scene on the second image sensor; and a processing stage for using the sensor output from the first imaging stage as the captured image signal and producing the output video image of the scene, said processing stage using the sensor output from the second imaging stage to detect a change in focus of the video image of the scene and to generate the focus change detection signal for the first imaging stage, whereby the focus change detection signal is applied to the lens focus adjuster of the first imaging stage in order to continually readjust the focus of the first image providing the output for the captured image signal.
 11. The video camera as claimed in claim 10 wherein the processing stage further uses the sensor outputs from one or both of the first and second imaging stages in order to initially focus the first and second images on the first and second image sensors.
 12. The video camera as claimed in claim 11 wherein the processing stage employs a “hill climb” contrast comparison technique upon the sensor outputs from the first and second imaging stages in order to initially focus the first and second images.
 13. The video camera as claimed in claim 10 wherein the processing stage employs a “hill climb” contrast comparison technique upon the sensor output from the second imaging stage to detect a change in focus of the video image of the scene and generate the focus change detection signal for the first imaging stage in order to refocus the first image.
 14. The video camera as claimed in claim 10 wherein the processing stage employs a continuous “whole way” auto focus technique upon the sensor output from the second imaging stage to detect a change in focus of the video image of the scene and generate the focus change detection signal for the first imaging stage in order to refocus the first image.
 15. The video camera of claim 10 wherein the processing stage further uses the sensor outputs from both the first and second imaging stages in a rangefinder configuration to detect a change in focus of the video image of the scene and to generate the focus change detection signal for the first imaging stage in order to refocus the first image.
 16. The video camera as claimed in claim 10 wherein the second imaging stage includes a second lens focus adjuster for continually readjusting focus of the second lens responsive to the focus change detection signal, and wherein the processing stage either (a) selects the sensor output from the first imaging stage as the captured image signal and uses the sensor output from the second imaging stage to generate the focus change detection signal for the selected imaging stage or (b) selects the sensor output from the second imaging stage as the captured image signal and uses the sensor output from the first imaging stage to generate the focus change detection signal for the selected imaging stage, whereby the focus change detection signal is applied to the lens focus adjuster of the selected imaging stage in order to readjust the focus of the image providing the captured image signal.
 17. The video camera as claimed in claim 10 wherein at least one of the first and second lenses is a zoom lens.
 18. A video camera for producing an output video image of a scene from a captured image signal, said video camera comprising: a first imaging stage comprising a first image sensor for generating a first sensor output, a first lens for forming a first image of the scene on the first image sensor, and a first lens focus adjuster for continually readjusting focus of the first lens responsive to a focus change detection signal; a second imaging stage comprising a second image sensor for generating a second sensor output and a second lens for forming a second image of the scene on the second image sensor; and a processing stage for using the sensor output from the first imaging stage as the captured image signal and producing the output video image of the scene, said processing stage using the sensor outputs from the first and second imaging stages in a rangefinder configuration to detect a change in focus of the video image of the scene and to generate the focus change detection signal, whereby the focus change detection signal is applied to the lens focus adjuster of the first imaging stage in order to continually readjust the focus of the first image providing the captured image signal.
 19. The video camera as claimed in claim 18 wherein the processing stage uses the sensor output from the first imaging stage to initially focus the first image.
 20. The video camera as claimed in claim 19 wherein the processing stage employs a “hill climb” contrast comparison technique upon the sensor output from one or both of the imaging stages to initially focus the first image.
 21. The video camera as claimed in claim 18 wherein the second imaging stage includes a second lens focus adjuster for continually readjusting focus of the second lens responsive to the focus change detection signal, and wherein the processing stage either (a) selects the sensor output from the first imaging stage as the captured image signal and uses the sensor outputs from the first and second imaging stages in the rangefinder configuration to generate the focus change detection signal for the first imaging stage or (b) selects the sensor output from the second imaging stage as the captured image signal and uses the sensor outputs from the first and second imaging stages in the rangefinder configuration to generate the focus change detection signal for the second imaging stage, whereby the focus change detection signal is applied to the lens focus adjuster of the selected imaging stage in order to readjust the focus of the image providing the captured image signal.
 22. The video camera as claimed in claim 18 wherein at least one of the first lens and the second lens is a zoom lens. 